Skip to content

feat(api): add POST /contracts/:id/refresh endpoint to force manual spec re-fetch - #2

Open
deborahairtel887-dotcom wants to merge 44 commits into
mainfrom
feat/258-refresh-contract-interface
Open

deborahairtel887-dotcom wants to merge 44 commits into
mainfrom
feat/258-refresh-contract-interface

Conversation

@deborahairtel887-dotcom

Copy link
Copy Markdown
Owner

Description

This PR implements the POST /contracts/:id/refresh endpoint to allow operators to manually trigger an immediate re-fetch of a contract's on-chain interface spec from Soroban RPC without requiring an indexer restart.

Changes Implemented

  • Added POST /contracts/:id/refresh: Auth-protected endpoint in crates/lumenqraph-api/src/routes/contracts.rs that validates the contract ID, clears the in-memory spec cache entry, re-fetches the contract WASM from Soroban RPC, parses the contractspecv0 section, updates contract_specs in the database, and records the spec version in contract_spec_versions.
  • Structured Error Handling: Returns the refreshed interface JSON on success, 400 Bad Request for invalid contract IDs or unparseable WASM specs, 404 Not Found for nonexistent contracts or Stellar Asset Contracts (SACs), and structured errors on RPC failure.
  • RPC Client Support: Added get_contract_wasm and get_ledger_entry methods to RpcClient in crates/lumenqraph-api/src/rpc.rs using stellar-xdr to retrieve contract instance and code entries.
  • SpecCache Invalidation: Added invalidate(contract_id) method to SpecCache in crates/lumenqraph-api/src/specs.rs to evict cached interface entries.
  • Routing & OpenAPI: Registered the route under the protected router in crates/lumenqraph-api/src/routes/mod.rs and documented it in OpenAPI specs (crates/lumenqraph-api/src/openapi.rs).
  • Documentation: Documented POST /contracts/:id/refresh in docs/API.md.

Closes Lumen-Scribe#258

ALLEN-AYODEJI and others added 30 commits August 30, 2026 08:28
…): dependabot cargo groups, webhook queue-depth metric, disaster recovery + DB invariant docs

Resolves four operability/observability gaps.

Lumen-Scribe#288 — Database invariants doc (docs/ARCHITECTURE.md)
Add a "Database Invariants" section that gathers the four load-bearing schema
constructs a contributor can break silently: the trg_update_contract_summary
trigger on events, the monotonic events.seq BIGSERIAL webhook watermark column,
the single-row webhook_state watermark table (forward-only, advanced inside the
enqueue transaction), and the single-row indexer_cursor resume sentinel. Each
row states the invariant, why it exists, and what fails at runtime if a
migration violates it.

Lumen-Scribe#284 — Disaster recovery procedure (docs/DEPLOYMENT.md)
Add a "Disaster Recovery" section. Covers restoring from a pg_dump / PITR backup
(the recommended path), rebuilding the derived index from scratch when there is
no backup (START_LEDGER=0 for the ~7-day RPC retention window, backfill for a
recent gap, deep-backfill from a data-lake export for older history), an
idempotency caveat about events.seq restarting on a new database lineage, a
table of what is permanently lost (api_keys, webhook_subscriptions, the delivery
queue and its watermarks, pre-retention events), and how to re-register webhook
subscriptions afterward without triggering a delivery storm (register before the
indexer starts, or set starting_seq to the current max(seq)).

Lumen-Scribe#283 — Webhook pending-delivery queue-depth metric
The dispatcher processed webhook_deliveries but exposed no gauge for the pending
backlog, so a dispatcher falling behind a slow subscriber was invisible.
- dispatcher.rs: add a process-global PENDING_DELIVERIES atomic and
  refresh_pending_gauge(), which runs
  SELECT COUNT(*) FROM webhook_deliveries WHERE status = 'pending'.
- main.rs: refresh it once per service tick.
- metrics.rs: emit lumenqraph_webhooks_pending_deliveries unconditionally from
  the atomic, so queue depth stays visible even if the scrape-time queries fail.
- monitoring/prometheus_alerts.yml: add LumenqraphWebhookBacklogHigh (>500 for
  5m, warning) and LumenqraphWebhookBacklogCritical (>5000 for 5m, critical).
- monitoring/grafana_dashboard.json: add a "Webhook Delivery Backlog" timeseries
  panel with 500/5000 threshold lines.
- docs/DEPLOYMENT.md, monitoring/README.md: document the new metric and alerts.

Lumen-Scribe#281 — Dependabot cargo grouping + enforced supply-chain check
- .github/dependabot.yml: split the catch-all cargo group into a security-updates
  group (unbatched, so a vulnerable transitive dep is never held back), a
  dedicated "stellar" group for stellar-*/soroban-* (bumps can touch the XDR /
  strkey decode path), and the batched everything-else group.
- .github/workflows/ci.yml: pin cargo-deny to the 0.14 line (still accepts the
  committed deny.toml schema), add build caching, and make
  `cargo deny check advisories` blocking so a known RUSTSEC advisory now fails
  CI between the weekly Dependabot runs. Licenses/bans/sources stay
  non-blocking pending a known-clean baseline.
…): SDK iterator cancellation, core hex error, async-graphql range pin, GraphQL transfer filters

Lumen-Scribe#279 — TypeScript SDK: paginateEvents iterator now supports cancellation
- The `paginateEvents` async generator wraps its page-fetch loop in
  `try`/`finally`. It creates an internal `AbortController` (`pageAborter`)
  chained to the caller's optional `signal`, and hands `pageAborter.signal`
  to every `eventsPage` call instead of the raw external signal.
- Breaking out of the `for await` loop early (or throwing) now runs the
  `finally`, which aborts `pageAborter` — so an in-flight `eventsPage`
  request is torn down instead of left running with its result discarded.
- The external `signal` is bridged both ways: an already-aborted signal
  short-circuits before the first request; an abort mid-iteration ends
  pagination on the next `.next()` with an `AbortError`. The external
  listener is removed in the `finally`, so nothing leaks onto a long-lived
  caller signal.
- Added three tests to `sdk/typescript/src/index.test.ts`: an early break
  aborts the signal handed to the page fetch and issues no further page; a
  caller abort mid-iteration stops pagination; an already-aborted signal
  makes no request at all.
- Files: sdk/typescript/src/index.ts, sdk/typescript/src/index.test.ts

Lumen-Scribe#280 — lumenqraph_core::Error gains a hex::FromHexError conversion
- Added `Hex(#[from] hex::FromHexError)` with
  `#[error("hex decode error: {0}")]`, alongside the existing `sqlx::Error`
  and `serde_json::Error` `#[from]` variants. `hex` is already a dependency
  of `lumenqraph-core`, so the manifest was left unchanged.
- `?` on a `hex::decode(...)` result now produces a typed `Error::Hex`
  instead of a lossy `.map_err(|e| Error::Other(e.to_string()))`.
- `crates/lumenqraph-core/src/spec.rs` and `src/read.rs` were checked: in
  the current code neither decodes hex through `lumenqraph_core::Error`
  (`spec` operates on `&[u8]`; `read::decode_hex` returns the crate-local
  `EncodeError`), so there were no manual conversions to remove there. The
  new variant is in place for those paths and for the indexer / api / mcp
  callers that hex-decode `spec_section`.
- Files: crates/lumenqraph-core/src/error.rs

Lumen-Scribe#282 — async-graphql: a range constraint instead of an exact pin
- `async-graphql` / `async-graphql-axum` change from `=7.0.7` to
  `>=7.0.7, <7.0.11`. Patch-level fixes in the 7.0.7..=7.0.10 window (still
  on axum 0.7) are now picked up automatically, while the 7.0.11 jump to
  axum 0.8 stays gated so it cannot be pulled in piecemeal.
- The manifest comment is expanded to explain the ceiling and to record
  that lifting `<7.0.11` is the async-graphql half of a workspace-wide
  axum 0.8 migration and should be done together with it.
- The committed `Cargo.lock` (async-graphql 7.0.7) still satisfies the new
  range, so the lockfile is unchanged.
- Files: Cargo.toml

Lumen-Scribe#285 — GraphQL `transfers` gains `from` / `to` address filters
- The resolver now accepts optional `from: String` and `to: String`
  arguments, mirroring the REST `GET /contracts/:id/transfers`
  `?from=` / `?to=` parameters: exact match on `from_addr` / `to_addr`
  through the same `($n::text IS NULL OR col = $n)` predicate form.
- SQL placeholders renumbered: $1 contract, $2 from, $3 to, $4/$5 keyset
  cursor, $6 limit.
- The SDL assertion test now also requires `from: String` and `to: String`
  to appear in the generated schema (the `Transfer` type's own fields are
  `fromAddr` / `toAddr`, so those substrings are unambiguous).
- Files: crates/lumenqraph-api/src/graphql.rs

Verification
- TypeScript SDK: `npm run typecheck`, `npm run lint`, and `npm test`
  (vitest) all pass — 55 tests, including the 3 new cancellation tests.
- Rust: the workspace does not build at this branch's base, for reasons
  unrelated to these changes — a non-existent `sqlx` "offline" feature in
  Cargo.toml and ~29 pre-existing compile errors in lumenqraph-indexer /
  lumenqraph-api. Those were left untouched. The `Error::Hex` variant was
  compiled in isolation; the graphql.rs and Cargo.toml changes are
  self-contained.
…): Retry-After header, smoke-test gating, Postgres volume docs, backfill RPC timeout

## Lumen-Scribe#290 — 429 responses now send a `Retry-After` header

`ApiError` gained a dedicated `RateLimited { retry_after_secs, message }` variant.
`ApiError::too_many_requests(Option<u64>)` now takes the computed wait time, and
`ApiError::into_response` emits a `Retry-After: <secs>` header whenever that value
is present. `rpc_auth_and_rate_limit` threads `RateLimitStatus::retry_after_secs`
into the error so RPC-route 429s carry the header too (the main
`auth_and_rate_limit` path already set it manually). SDK clients that read
`Retry-After` can now back off precisely instead of always falling through to
exponential backoff.

Tests: unit tests in `error.rs` assert the header is present with the exact
value for a rate-limited error, absent when no hint is supplied, and absent on
non-rate-limited errors; a new Postgres-backed integration test in `auth.rs`
asserts a live 429 response carries an integer `Retry-After`.

Files: crates/lumenqraph-api/src/error.rs, crates/lumenqraph-api/src/auth.rs

## Lumen-Scribe#289 — Smoke test gated behind a cargo feature

`crates/lumenqraph-indexer/src/smoke.rs` (a heavy end-to-end pipeline test —
mock RPC, no live network, but needs Postgres) is now gated behind
`#[cfg(all(test, feature = "smoke-tests"))]` in addition to `#[ignore]`. Without
the `smoke-tests` feature the module is not compiled at all, so it can never run
in an offline CI job that omits `TEST_DATABASE_URL`. Added a `smoke-tests`
feature to the indexer crate, a `make test-smoke` target, a dedicated
"Test (smoke, end-to-end)" CI step that runs it explicitly where Postgres is
available, and a "Smoke tests" section in CONTRIBUTING.md.

Note: the smoke test uses an in-process mock Soroban RPC, so the original
report's "live RPC calls" premise does not hold today; the feature gate still
delivers the requested guarantee that it cannot run by accident.

Files: crates/lumenqraph-indexer/Cargo.toml, crates/lumenqraph-indexer/src/main.rs,
crates/lumenqraph-indexer/src/smoke.rs, Makefile, .github/workflows/ci.yml,
CONTRIBUTING.md

## Lumen-Scribe#287 — Document Postgres volume management

`docker-compose.full.yml` already mounts a named `pgdata` volume at
`/var/lib/postgresql/data` with a top-level `volumes:` declaration, so data
already survives `docker compose down`. This change adds the missing operational
documentation: a "Postgres data volume" section in docs/DEPLOYMENT.md covering
the volume name, what does and does not delete it, `pg_dump`/`pg_restore` backup
and restore, and a raw volume-archive alternative.

Files: docs/DEPLOYMENT.md

## Lumen-Scribe#286 — backfill.sh RPC timeout override

`scripts/backfill.sh` now accepts `--rpc-timeout <secs>` (and
`--rpc-timeout=<secs>`), exporting `RPC_TIMEOUT_SECS` before invoking the indexer
so it overrides any `.env` value. Added a header comment explaining why archive
RPCs need a higher value (a timeout fails the whole batch and the retry reuses
the same timeout), an "Archive RPC timeouts" section in docs/DEEP_BACKFILL.md,
and a pointer in .env.example. Argument parsing preserves the existing positional
`<start_ledger>` usage.

Files: scripts/backfill.sh, docs/DEEP_BACKFILL.md, .env.example
…umen-Scribe#210-213)

- Lumen-Scribe#210: checkpoint cursor after each backfill page so interrupted runs are
  resumable; re-running with from_ledger=0 continues from the last stored page
- Lumen-Scribe#211: reject from > to in GET /contracts/:id/interface/diff with 400
- Lumen-Scribe#212: add MAX_REQUEST_BODY_BYTES env var (default 65536); keep API_MAX_BODY_BYTES
  as a backward-compat alias
- Lumen-Scribe#213: add METRICS_REQUIRE_API_KEY env var (default false) to gate GET /metrics
  behind the same API key middleware as data routes

Tests added for Lumen-Scribe#210 (backfill_cursor_checkpointed_per_page) and Lumen-Scribe#211
(diff param boundary conditions). Docs updated in .env.example and README.
…in-one.sh

Add an explicit 'sqlx migrate run' step for both the primary and optional
testnet databases before any service process is forked. If migrations fail
the script exits immediately instead of starting a partially-configured
stack. The indexer is launched with SKIP_MIGRATIONS=true to avoid a
redundant second run.
Add migration 0022 that enforces CHECK (kind IN ('transfer', 'mint',
'burn', 'clawback')) at the database level. Application-level enforcement
in store::extract_transfer is correct, but an unconstrained TEXT column
allows a bug or direct DB write to insert rows with unexpected kind values
that would be silently served by the API.
…, CI job

- pyproject.toml: PEP 517 build config with zero runtime deps; pytest +
  pytest-asyncio + mypy as dev extras
- async_client.py: AsyncLumenqraphClient mirroring the sync surface;
  I/O via run_in_executor (no third-party deps); paginate_events and
  paginate_transfers as async generators
- webhook.py: fix verify_webhook_signature to require the sha256= prefix
  matching the server's X-Lumenqraph-Signature header format
- __init__.py: export AsyncLumenqraphClient
- tests: rewritten webhook tests with correct sha256= fixtures; new async
  client unit tests covering URL building, cursor pagination, errors, and
  the async context manager protocol
- ci.yml: new sdk-python job running mypy + pytest on Python 3.8 and 3.12
…bot docker+pip

Pin builder and runtime stages to their current SHA256 digests so that
builds from the same commit always pull identical layers:

  rust:1-slim@sha256:17d1ba895198f9934c6314ec5346a0d5115372f3243390c3d731e242f35c2f27
  debian:bookworm-slim@sha256:88200866dfff7ea7f5cbcb6ec7c8a701889efe6fe859fe64d6990e4b07ea4171

Add 'docker' and 'pip' ecosystems to .github/dependabot.yml so digest
pins and Python dev-deps are refreshed automatically on the weekly
Monday schedule alongside the existing cargo/npm/github-actions entries.
Lumen-Scribe#218 — Circuit breaker for repeated RPC failures
- Add MAX_CONSECUTIVE_ERRORS (default 20) and DEGRADED_POLL_INTERVAL_SECS
  (default 300) to indexer Config
- Poller tracks consecutive_errors; after threshold, logs at ERROR level
  and switches to the degraded sleep interval instead of normal backoff
- Counter resets on the first successful cycle
- cursor::set_consecutive_errors() writes the current count to a new
  consecutive_errors BIGINT column in indexer_cursor
- migration 0022_circuit_breaker_gauge.sql adds the column
- API /metrics exposes lumenqraph_consecutive_errors gauge

Lumen-Scribe#219 — Centralise CONTRACT_IDS validation in lumenqraph-core
- Add parse_contract_ids() to lumenqraph-core/src/xdr.rs (validates
  C-strkey format and ≤25-ID RPC limit); re-export from lib.rs
- Indexer config.rs delegates to core instead of inline logic
- API, webhooks, and MCP main.rs call parse_contract_ids() at startup
- Unit tests for parse_contract_ids in xdr.rs

Lumen-Scribe#220 — WEBHOOK_ENCRYPTION_KEY read once at startup, not per delivery
- Add encryption_key: String to webhooks Config; Config::from_env()
  reads and validates it (fails fast if absent or empty)
- fetch_due() accepts &str encryption_key instead of env::var per call
- deliver() passes &config.encryption_key through
- Removed redundant per-startup checks from webhooks/api main.rs
- Fixed 3 test call sites; added unit test in config.rs

Lumen-Scribe#221 — MCP JSON-RPC protocol integration tests (stdio round-trip)
- Refactor serve() into serve_io<R, W>() generic over AsyncRead/AsyncWrite
- serve() calls serve_io with stdin/stdout (unchanged production path)
- New protocol_tests module drives serve_io with tokio::io::duplex streams
- Tests cover: initialize, tools/list, tools/call round-trip, malformed
  JSON (parse error -32700), unknown method (-32601), notifications
  (no response), missing required arg (isError), unknown tool name
…r.yaml

Closes Lumen-Scribe#250.

render.yaml was missing WEBHOOK_ENCRYPTION_KEY entirely, meaning any
Blueprint deploy would silently fall back to the hardcoded
'default-key-for-testing' value. It also had no DATABASE_MAX_CONNECTIONS
entry, leaving SQLx's pool unconfigured against Supabase's 60-connection
free-tier ceiling.

Changes:
- render.yaml: add WEBHOOK_ENCRYPTION_KEY with sync:false (Render will
  prompt the operator to set it before deploy; no default is shipped).
- render.yaml: add DATABASE_MAX_CONNECTIONS=10 — a sensible combined
  ceiling for the all-in-one indexer+api container against Supabase free
  (3 indexer + 8 api, with headroom for migrations and the Supabase pooler).
- docs/DEPLOYMENT.md: insert step 2 in the Render walkthrough instructing
  operators to generate the key with 'openssl rand -hex 32' before running
  the Blueprint.
- docs/DEPLOYMENT.md: add WEBHOOK_ENCRYPTION_KEY to the Production
  Checklist with a warning against using the default key.
- docs/DEPLOYMENT.md: add a Render+Supabase callout under Connection Pool
  Sizing explaining the DATABASE_MAX_CONNECTIONS=10 choice and when to
  raise it.
)

Closes Lumen-Scribe#251.

CHANGELOG.md contained only high-level feature summaries with no
structured guidance for operators upgrading a running deployment:
no breaking-change notices, no migration steps, no env var changelog.

Changes:
- docs/UPGRADING.md (new): structured upgrade guide covering every
  release, modelled on the format requested in the issue:
    * Breaking changes with required action (env vars, service stop order)
    * Database migration log (0001-0021) with a description of each
    * New/renamed environment variable table per release
    * Recommended upgrade procedure with explicit shell commands
    * General notes on how migrations work, upgrade ordering, health
      verification, and rollback options
  Sections:
    - 'Unreleased → next release' covering the 11 pending migrations
      and 5 breaking changes (WEBHOOK_ENCRYPTION_KEY required, CORS
      default tightened, GraphQL introspection off, token_transfers.kind
      added, webhook delivery schema change)
    - 'Fresh install / v0.1.0' covering the 9 initial migrations and
      minimum required env vars
    - 'General notes' on migration mechanics, ordering, and rollback

- CHANGELOG.md: added 'Migration notes' sub-section to [Unreleased]
  and [0.1.0] entries with a summary of breaking changes and a
  cross-reference to the new UPGRADING.md; added a top-level callout
  pointing operators to UPGRADING.md before any upgrade.

- README.md: added 'Upgrading Between Versions' link to the table of
  contents; updated the 'Running in production' section to direct
  operators to UPGRADING.md; updated the Contributing section footer
  to reference both CHANGELOG.md and UPGRADING.md.
…webhook key, MCP protocol tests

Lumen-Scribe#218 - Circuit breaker for repeated RPC failures
- Add MAX_CONSECUTIVE_ERRORS (default: 20) and DEGRADED_POLL_INTERVAL_SECS
  (default: 300s) config fields in lumenqraph-indexer
- Poller enters degraded state after N consecutive failures, sleeping for
  the degraded interval instead of cycling through max backoff indefinitely
- Emits ERROR-level log when circuit opens; resets on first successful cycle
- Exposes lumenqraph_consecutive_errors Prometheus gauge via cursor table
- Unit tests for open/close/reset/disabled-when-zero behaviour

Lumen-Scribe#219 - CONTRACT_IDS validation in all services
- lumenqraph-core::parse_contract_ids already validates C-strkeys and
  enforces the 25-ID RPC limit; all four services call it at startup
- Add contract_ids_startup_validation test module to lumenqraph-api,
  lumenqraph-webhooks, and lumenqraph-mcp covering: G-strkey rejection,
  garbage input, too-many-IDs, empty string, valid C-strkey, mixed input

Lumen-Scribe#220 - Webhook encryption key read once at startup
- Move WEBHOOK_ENCRYPTION_KEY out of the per-tick env lookup into
  Config::encryption_key, read once in Config::from_env
- Hard fail-fast at startup if the key is absent or empty (no silent
  fallback to a test default)
- Pass the key through to fetch_due and deliver via the Config struct

Lumen-Scribe#221 - MCP JSON-RPC protocol layer integration tests
- Add protocol_tests module to lumenqraph-mcp::main driven by serve_io
  over in-process tokio::io::duplex streams (no real DB or stdio needed)
- Covers: initialize round-trip, tools/list, full 3-step handshake,
  malformed JSON (-32700), unknown method (-32601), notifications
  (no response), empty lines, ping, missing required tool arg (isError),
  unknown tool name (isError), multiple independent messages
…): webhook HMAC slices, dashboard metrics CI, contract import resilience, e2e testing

Lumen-Scribe#242: Fix verifyWebhook to handle Uint8Array views that slice a larger buffer
- Replace rawBody.buffer cast with buffer.slice(byteOffset, byteOffset + byteLength)
- Ensures HMAC is computed over exactly the view's bytes, not the entire backing buffer
- Prevents false negatives when Node.js Buffer instances share underlying ArrayBuffer

Lumen-Scribe#243: Add CI validation for Grafana dashboard metrics
- Create validate_dashboard_metrics.py to extract metrics from dashboard JSON
- Compare dashboard references against metrics defined in Rust crates/*/src/metrics.rs
- Add new CI job 'dashboard-metrics' to catch drift early, preventing silent panel failures
- Script warns on unused metrics in code but fails on missing/undefined references

Lumen-Scribe#244: Enhance import_contracts.py with error handling and resilience
- Add per-request retry logic with exponential backoff for network failures
- Validate contract ID format (C-strkey pattern) before import attempts
- Collect and report failed imports with reasons instead of silent failures
- Add --api-url option to import contracts directly into Lumenqraph instance
- Exit with code 1 if any imports fail (was silently succeeding before)
- Configurable timeout and max retry attempts

Lumen-Scribe#245: Add end-to-end test for full indexer → database → API pipeline
- Create e2e_smoke_test.sh that verifies health endpoints and queries contracts API
- Add e2e-test.yml workflow running nightly and on-demand via workflow_dispatch
- Start full stack via docker-compose.full.yml with test environment
- Test suite verifies:
  - API health endpoint becomes healthy
  - /contracts endpoint returns valid JSON
  - /health endpoint has expected fields
  - GraphQL endpoint is reachable
- Collect and log service output on test failure for debugging

Closes Lumen-Scribe#242 Closes Lumen-Scribe#243 Closes Lumen-Scribe#244 Closes Lumen-Scribe#245
Fixes Lumen-Scribe#246: SDK codegen check now explicitly validates that TypeScript types
are regenerated from current openapi.yaml. Added clearer logging and error
messages to make the regeneration process explicit.

Fixes Lumen-Scribe#247: Added Prometheus metrics to CallCache to enable production tuning:
- lumenqraph_call_cache_hits_total: Total cache hits
- lumenqraph_call_cache_misses_total: Total cache misses
- lumenqraph_call_cache_evictions_total: LRU evictions when capacity exceeded
- lumenqraph_call_cache_size: Current entry count (gauge)

Fixes Lumen-Scribe#248: Enhanced webhook URL validation to prevent DNS rebinding SSRF attacks.
- Registration-time validation: blocks obvious internal/private addresses
- Delivery-time validation: re-validates by resolving hostname to verify IP is public
- Moved url_validation to lumenqraph-core for reuse across API and webhooks

Fixes Lumen-Scribe#249: Added optimistic locking to indexer_cursor to prevent concurrent
writer conflicts during rolling deployments:
- New migration adds version column
- write_progress() now uses WHERE version = $expected for atomic updates
- Logs warning and returns error if version mismatch detected
…): add TypeScript SDK event methods, spec concurrency limit, and pool configuration

Closes Lumen-Scribe#238 Lumen-Scribe#239 Lumen-Scribe#240 Lumen-Scribe#241

## Changes

### Lumen-Scribe#239: TypeScript SDK methods for fetching events
- Add `getEvent(eventId: string)` to fetch a single event by ID
- Add `getTransactionEvents(txHash: string, opts?: { limit?: number })` to list events for a transaction
- Both methods available on LumenqraphClient for typed SDK consumers

### Lumen-Scribe#240: Indexer spec fetch concurrency control
- Add SPEC_FETCH_CONCURRENCY config (default: 4) to bound simultaneous RPC calls during spec fetching
- Wrap spec fetches in tokio::sync::Semaphore; cached specs bypass the limit entirely
- Prevents rate limiting and connection exhaustion during large catch-up cycles with many new contracts
- Update all SpecCache instantiation sites (poller, backfill, deep_backfill, reenrich, smoke tests)

### Lumen-Scribe#241: Database connection pool size configuration
- Add DATABASE_MAX_CONNECTIONS config (default: 10) for explicit pool size control
- Add DATABASE_MIN_CONNECTIONS config (default: 0) for idle connection maintenance
- Apply pooling settings in indexer's main.rs; API and webhooks already read these via env
- Document sizing guidance in .env.example for free-tier vs production deployments

### Lumen-Scribe#238: Postgres health check
- Verify docker-compose.yml and docker-compose.full.yml have health checks configured
- Both files already include: `pg_isready -U lumenqraph` health check on postgres service
- docker-compose.full.yml already has depends_on: { postgres: { condition: service_healthy } }

## Documentation
- Add detailed comments in .env.example for all new environment variables
- Include sizing guidance for different deployment scenarios (free-tier, production)
- Document trade-offs for DATABASE_MIN_CONNECTIONS and SPEC_FETCH_CONCURRENCY
): Add webhook SDK methods, fix credential exposure, document multi-network, add reenrich progress

Lumen-Scribe#230: TypeScript SDK webhook management
- Add Webhook, WebhookDelivery, CreateWebhookOptions, UpdateWebhookOptions types
- Implement createWebhook(), listWebhooks(), deleteWebhook(), updateWebhook(), listDeliveries() methods
- Type-safe webhook subscription management without hand-rolling HTTP calls
- Add private delete() helper for DELETE requests

Lumen-Scribe#231: Security fix in gen_api_key.sh
- Remove database password from command-line arguments (visible in /proc/*/cmdline and ps)
- Use PGPASSWORD environment variable instead of embedding in DATABASE_URL
- Document .pgpass file usage for secure password handling
- Update usage documentation

Lumen-Scribe#232: Document multi-network proxy feature
- Add "Multi-network deployments" section to README with INSTANCE_MOUNTS explanation
- Link to docs/MULTI_NETWORK.md for detailed patterns and configuration
- Add INSTANCE_MOUNTS to Configuration table with examples
- Add note to API table explaining multi-network path prefixes
- Add INSTANCE_MOUNTS to .env.example with comments and examples

Lumen-Scribe#233: Add progress reporting to reenrich command
- Count total events upfront for accurate progress estimation
- Report progress every 10,000 events with throughput (events/sec)
- Estimate remaining time based on current throughput
- Print progress bar to stderr when running in TTY
- Include elapsed time, processed count, and ETA in logs and terminal output
- Log structured info records for monitoring and analysis

Closes Lumen-Scribe#230 Lumen-Scribe#231 Lumen-Scribe#232 Lumen-Scribe#233
…): XDR unknown types, contracts feature_disabled, request-id header, typed GraphQL params

Closes Lumen-Scribe#234 Lumen-Scribe#235 Lumen-Scribe#236 Lumen-Scribe#237

Lumen-Scribe#234 - XDR decoder unknown ScVal types
- Change fallback for unknown XDR tags to return structured {"_type": "unknown", "xdr_tag": tag}
- Change fallback for malformed XDR to return structured {"_type": "unknown", "xdr": base64}
- Add test for unknown ScVal tag discrimination
- Update existing malformed test to match new format

Lumen-Scribe#235 - Contracts API feature_disabled error
- Add FeatureDisabled error code to ApiError enum
- Add feature_disabled() helper method returning 501 Not Implemented
- Update contract_state to detect if state indexing is disabled
- Update contract_data to detect if key indexing is disabled
- Return feature_disabled when no data exists in entire table, not_found for specific contract

Lumen-Scribe#236 - Request ID in response headers
- Import request_id module in routes/mod.rs
- Add request_id_middleware to router middleware chain
- Middleware already sets X-Request-ID in responses, now properly integrated

Lumen-Scribe#237 - GraphQL typed params for enriched events
- Add EnrichedParam SimpleObject with name, type, value fields
- Convert Event from SimpleObject to Object to support custom resolvers
- Add params field resolver that extracts params from enriched JSON
- Clients can now query individual params with field selection
…be#264 Lumen-Scribe#265 Lumen-Scribe#266

fix(indexer): use Acquire ordering in take_metrics for visibility on ARM (Lumen-Scribe#265)
feat(api): add cursor pagination to GET /contracts with next_cursor (Lumen-Scribe#266)
feat(api): add POST /webhooks/:id/rotate-secret with grace period (Lumen-Scribe#264)
feat(core): accept M-strkey muxed accounts as source_account in read layer (Lumen-Scribe#263)
docs: document webhook secret rotation in SECURITY.md and docs/API.md
docs: document MuxedAccount support in docs/API.md
sdk(ts): update listContracts to return ContractsResponse with pagination

Closes Lumen-Scribe#263
Closes Lumen-Scribe#264
Closes Lumen-Scribe#265
Closes Lumen-Scribe#266
Closes Lumen-Scribe#255.

Problem: POST /contracts/:id/simulate (and /call) returns spec_unavailable
(404) for both unindexed contracts (retry later) and Stellar Asset Contracts
(retrying will never help, there is no WASM spec). Callers had no way to
distinguish the two cases from the error code.

Changes:
- error.rs: Add SacNotSupported variant (HTTP 422) to ErrorCode enum and
  a matching ApiError::sac_not_supported() constructor.
- specs.rs: Return sac_not_supported when:
    1. No contract_specs row exists but events do (confirmed SAC), and
    2. A contract_specs row exists but spec_section is empty/null (SAC or
       equivalent with no callable interface).
  Return spec_unavailable (404) only for the genuine "not indexed yet,
  retry later" case.
- routes/read.rs: Fix the contract_not_in_cache_returns_404 test, which
  was constructing an incomplete AppState struct literal (missing fields
  added in later PRs), and add the missing AppState fields to make_state.
- docs/API.md: Document both sac_not_supported (422) and the clarified
  spec_unavailable (404) semantics.
Closes Lumen-Scribe#254.

Problem: scripts/benchmark_indexer.sh timed indexer throughput end-to-end
including Soroban RPC latency. Network conditions vary widely (10-300 ms/page)
and completely dominate the measurement, making throughput numbers
non-comparable across runs and unsuitable for regression detection.

Changes:
- scripts/benchmark_indexer.sh: Rewritten to invoke the new Rust criterion
  benchmarks instead of running the indexer binary against a live RPC.
  Supports --phase (xdr_decode | enrichment | db_insert | all), --baseline
  FILE, and --save-baseline FILE for regression gating. Exits non-zero when
  any phase regresses by more than 10% vs the baseline.
- crates/lumenqraph-indexer/benches/bench_indexer.rs: New criterion benchmark
  with three isolated phases:
    * xdr_decode: Base64 XDR -> JSON decode throughput (CPU-only, no I/O)
    * enrichment: spec-driven named/typed enrichment throughput (CPU-only)
    * db_insert: UNNEST batch INSERT into Postgres (gated on TEST_DATABASE_URL)
  Uses synthetic in-process data for CPU phases and the same UNNEST query as
  store::insert_events for the DB phase -- no live RPC, no network jitter.
- crates/lumenqraph-indexer/Cargo.toml: Add criterion 0.5 (async_tokio) as a
  dev-dependency and declare the [[bench]] harness.
- Cargo.toml: Add criterion 0.5 to workspace dependencies.
- docs/BENCHMARKING.md: Rewritten to document the three-phase structure,
  how to run each phase, expected baseline numbers, and how to use the
  --baseline flag for CI regression detection.
…4-288-dependabot-webhook-queue-metric-disaster-recovery-db-invariants

fix(Lumen-Scribe#281,Lumen-Scribe#283,Lumen-Scribe#284,Lumen-Scribe#288): dependabot cargo groups, webhook queue-depth metric, disaster recovery + DB invariant docs
…282-285-sdk-cancellation-hex-error-async-graphql-pin-graphql-transfers-filters

fix(Lumen-Scribe#279,Lumen-Scribe#280,Lumen-Scribe#282,Lumen-Scribe#285): SDK iterator cancellation, core hex error, async-graphql range pin, GraphQL transfer filters
…9-290-retry-after-smoke-gating-pgdata-backfill-timeout

fix(Lumen-Scribe#286,Lumen-Scribe#287,Lumen-Scribe#289,Lumen-Scribe#290): Retry-After header, smoke-test gating, Postgres volume docs, backfill RPC timeout
…-211-212-213

fix: resumable backfill, diff validation, body limit, metrics auth (#…
…-250-render-yaml-security-defaults

fix: add WEBHOOK_ENCRYPTION_KEY and DATABASE_MAX_CONNECTIONS to render.yaml
…-251-upgrading-guide

docs: add structured upgrade and migration guide
Lost-Z added 14 commits August 31, 2026 09:50
…9-codegen-cache-ssrf-cursor

fix: SDK codegen, call cache metrics, SSRF validation, cursor locking (Lumen-Scribe#246 Lumen-Scribe#247 Lumen-Scribe#248 Lumen-Scribe#249)
…41-healthcheck-sdk-concurrency-pooling

fix(Lumen-Scribe#238,Lumen-Scribe#239,Lumen-Scribe#240,Lumen-Scribe#241): healthcheck, SDK methods, spec concurrency, pool config
…45-webhook-dashboard-contracts-e2e

fix(Lumen-Scribe#242,Lumen-Scribe#243,Lumen-Scribe#244,Lumen-Scribe#245): webhook HMAC slices, dashboard metrics CI, …
…-sdk-webhook-security-docs-progress

Add webhook SDK, fix credential exposure, document multi-network, add reenrich progress (Lumen-Scribe#230-Lumen-Scribe#233)
…7-xdr-contracts-request-id-graphql

fix: XDR unknown types, contracts feature_disabled, request-id header, typed GraphQL params
…code

fix(api): add sac_not_supported error code for Stellar Asset Contracts
…mock-rpc

fix(bench): isolate benchmark phases from network latency
…259-webhook-rate-limit-and-max-subscriptions

Enforce rate limiting and max subscription limit on POST /webhooks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

No API endpoint exists to trigger a manual re-fetch of a contract's on-chain interface, requiring a full redeployment to pick up a missed spec